Integrating third party services like a boss

How to integrate any third party service in your app, even if they don't provide any developer APIs.

As developers, we are accustomed to using third-party libraries, tools, services, plugins and whatever else we get our hands at, to build our applications. In this era of agile development and ever changing customer and business needs, we can't afford to build every solution from scratch on our own. It's neither feasible nor economical to do so, especially when we want to focus on our core business and not get lost in the implementation details of an unrelated domain. Take an example of an e-commerce platform, their core business is selling products by letting their customers buy and pay online. Now imagine them building their own payment solution, worrying about the plethora of banking and financial regulations from all the differents sources, banks, credit cards, etc. Unless they are making a sale of millions, if not billions, they would be better of using a third-party payment service with good API integrations to enable use of their payment services.

Companies use third-party services to build products with faster iterations, save cost and focus on their core business.

However, not all third-party services will provide an exhaustive list of APIs for us to seamlessly integrate the service in our platform. In most cases, the smaller the service and its customer base, the lesser they would care about providing a detailed developer APIs. It's not that uncommon that some of these services might not even have any consumable APIs and they would just provide their application UI to interact and use their services. I know, I know, at this point you might be thinking, why would we want to use such a service in the first place. But for a moment leave the developer brain behind and think of it from the business perspective. Maybe when your company was young, the needs might have been different, the service might just have been a tool for your company to perform some regular tasks, some bookkeeping, some totally unrelated action that doesn't necessarily have any touchpoint with your company's application or business needs. But as the business evolved and grew, there might have been need to get a few information from this tool for some data analytics and reporting purposes. It would be totally unreasonable to start using a new tool overnight, especially when all we needed was a few information to be extracted from this tool. So, as a developer, our challenge is to programmatically extract this relevant information available in the UI of the tool, without having any consumable APIs from the service. Let's first see what information we have at hand to tackle this problem.

Some background to facilitate the integration

• Every service would use some mechanism to authenticate the user, most likely it would be some httponly session cookies.
• There would be an internal url endpoint that the service's UI would use to set those cookies aka login the user.
• The information shown in the UI of the service, would most likely be fetched from some POST/GET requests to their internal APIs.

Steps

Having the above information in mind we would need to perform the following steps in order to programatically get the information we need form the service.

• Inspect the network calls of service's UI. We are particularly interested in the login endpoint and the endpoint for the page that shows the information that we are interested in. Though you may use the tool of your choice to perform the network sniffing, I recommend using the Network tab from the Chrome DevTools for an easier and straightforward view.
• Once we have identified the calls to their login and the data endpoints, and also the required request format and body, we need to first issue a login request using http request library in the programming language of your choice, and save all the cookies sent as part of the login response.
• Now we can make the subsequent requests to the data endpoints, passing the previously fetched cookies as part of the cookie headers of our requests.

For example if you're using JavaScript and Node.js for your backend, you may do something like below to create a service client that would mimic the requests to the service's internal endpoints.

• Fist up, install the libraries  npm install lodash request got
• Now create the client.js file with the following code to create a client that can be used to send requests to the service's internal APIs.

const _ = require('lodash')
const {promisify} = require('util')
const got = require('got')
const request = require('request')
const requestPromise = promisify(request)

const SERVICE_URL = 'https://www.someservice.com'
const email = '[email protected]'
const password = 'service_login_password'

// function to get cookies string
// that could be passed as cookie headers
async function getCookies() {
  const jar = request.jar()

  await requestPromise({
    jar,
    url: `${SERVICE_URL}/login/`,
    method: 'POST',
    // this will ensure that the redirects
    // are followed till it resolves
    // including `POST` to `GET` redirects
    followAllRedirects: true,
    // the exact body format may differ
    // it may be `json` for the service you're using
    form: {action: 'login', email, password},
  })

  return jar.getCookieString(SERVICE_URL)
}

// create a service client that can be used to make
// requests to the internal endpoints of the service
module.exports = got.extend({
  baseUrl: SERVICE_URL,
  hooks: {
    beforeRequest: [
      async options => {
        // if you are confident that the cookies are long lived
        // for the service that you use, you may also save and re-use
        // the same cookie headers in your application code
        // this will avoid the need to login to the service
        // each time a request is made to the service endpoints
        _.merge(options, {headers: {cookie: await getCookies()}})
      },
    ],
  },
})

• Once we have the service client, we can use this to make http requests to the internal endpoint of the service.

const client = require('./client')

async function getData() {
  const response1 = await client.post('some/internal/endpoint', {
    ...
    body: {...}
  })
  const response2 = await client.post('another/internal/endpoint', {
    ...
    body: {...}
  })
  ...

  console.log(response1.body, response2.body, ...)
}

And now we have a fully functional service client, without the actual service exposing any developer APIs.

While the above trick should work in normal use cases, of course it is not a silver bullet and you may have to include certain additional information depending upon the service you are trying to integrate. If the service's internal APIs are having additional checks and validations in place to ensure their APIs integrity, you have to do some more sniffing to find out what additional headers etc are needed to complete the requests. For example some of the services might be using CSRF token to protect their APIs, in which case you'd also need to get this information from their response headers and include the same in your request headers. Additionally if the response returned from their internal endpoints is not directly consumable data (eg it's a blob, html fragment, etc), you might need to parse that response for the information that you might be interested in. In either case, I hope by now you have gotten a fair understanding of the underlying idea behind this, and you'd be able to integrate any third-party service like a boss ;)

If you happen to run into issues or spot any mistakes with this article, please feel free to comment and I'd try my best to help you out / correct the mistakes.

Happy Coding!