2-3 Call a Web Service

This example introduces the GenerateFlowFile processor and demonstrates how to consume JSON data from a paged web service. All previous examples used the GetFile processor to start the flow. GenerateFlowFile is another way to start a flow when an event-driven flow is not applicable. On GenerateFlowFile's Scheduling tab we set the Run Schedule to "1 day" so that the data from the web service is only retrieved once per day. For testing purposes, every time you stop and start the GenerateFlowFile processor, it will create one FlowFile, initiating the flow.

After the initial call to the web service, we go into a loop where we get the paging parameters returned in the response ("page" and "total_pages"), check if there are more pages to fetch with a RouteOnAttribute processor and, if there are more pages, set the web service URL to get the next page.

For each response from the web service we split the JSON on the data array, look up an ID property in the JSON, construct the MarkLogic URI then finally insert into MarkLogic.

• Download Template
• Processors:
  • GenerateFlowFile – starts a flow on a timer
    • Scheduling
      • Run Schedule: 1 day
      • UpdateAttribute ("Set Initial Service URL")
    • Properties
      • service.url: https://reqres.in/api/users (custom property)
      • service.base.url: https://reqres.in/api/users (custom property)
  • InvokeHTTP ("Call Example Web Service")
    • Properties
      • Remote URL: ${service.url}
  • EvaluateJsonPath ("Get Response Paging Values")
    • Properties
      • servicePage: $.page (custom property)
      • serviceTotalPages: $.total_pages (custom property)
  • RouteOnAttribute ("Continue if More Pages")
    • Properties
      • hasMorePages: ${serviceTotalPages:gt( ${servicePage} )}
  • UpdateAttribute ("Set Paged Service URL")
    • Properties
      • service.url: ${"service.base.url":append('?page='):append( ${servicePage:plus(1)} )} 
  • SplitJson ("Split Response")
    • Properties
      • JsonPath Expression: $.data.*
  • EvaluateJsonPath ("Get ID from JSON")
    • Properties
      • json.id: $.id
  • UpdateAttribute ("Set MarkLogic URI")
    • Properties
      • marklogic.uri: /json/${json.id}.json
  • InvokeHTTP ("PUT to MarkLogic")
    • Properties
      • HTTP Method: PUT
      • Remote URL: http://localhost:8000/LATEST/documents?uri=${marklogic.uri}
      • Basic Authentication Username: youruser
      • Basic Authentication Password: yourpassword
    • Settings
      • Check all five checkboxes under "Automatically Terminate Relationships"