How to use the new Send HTTP Request action

Ben Regier

A few people have already posted about the new ‘Send an HTTP Request’ action and all the possibilities that it opens up for workflows in Power Automate. This action is basically a passport that allows you to access any of the APIs that your account has rights to - and not just Raiser's Edge!

But that power comes with a price – the HTTP action is not as easy to use as most of the other actions. However, everything you need to know to set up your HTTP request is included in the SKY API documentation, and it's not as hard as you might think.

I’m going to try to take you through the process of gathering the information you need from the SKY API documentation and putting it into your flow.

First of all, you can find the SKY API documentation here. The Power Platform connectors have so far only provided access to Raiser's Edge data, but the HTTP Request action can access any of these APIs. For our examples, let's try making some calls to Financial Edge.

First, we have to find our endpoint. I click on Financial Edge, and then go to the Accounts Payable section, and then click on Endpoint Reference. Then I can look through the list of endpoints to find the one I’m looking for.

To make a request, I will need to know these things:

• The method
• The request URL
• Whether there are any request parameters
• If I am creating or updating information, I need to know how to structure that information in the request body
• The response schema

Method

The method is right at the top of the page, and you can also see it on the list on the left-hand side as well. SKY API uses four different methods.

• Get requests are requests for information about something
• Delete requests are requests to delete something
• Post requests are requests to create something
• Patch requests are requests to update something

Looks like this is a Get request, so I set my HTTP action accordingly:

Request URL

Next, I need the Request URL. That can also be found near the top of the endpoint documentation page.

There are a few things you need to know about the request URL.

• The first part of the URL (https://api.sky.blackbaud.com) is added automatically by the connector, so we don't need to copy that.
• The URL may have curly brackets {like_this} or square brackets [like_this] around parts of it. These are request parameters, and we’ll talk more about them in a minute. For now, just be aware that they mean that you might need to add some data (like a record id) at that point in the URL.

I copy the relevant part of the URL into my HTTP action:

Request Parameters

Request parameters are pieces of information that you include in your request to let SKY API know what you want. A classic example of this is the ID. If you want to see an address, you need to pass the address ID as a request parameter so that SKY API knows which address to send you.

Another example would be filters. If you are requesting a list of invoices, you may be able to narrow your list by providing a date range, invoice status, etc.

Let’s look at a simple example first. This request URL has a vendor ID request parameter in middle of the URL. The curly brackets let you know that it’s required, and if you weren’t sure, you can check the Request parameters area of the documentation page to see that it is required and read a little description about what it is.

When you paste this request URL into your flow, you will have to replace those brackets with an ID, either by typing it in or by using dynamic content from another action.

Next, let's look at a different example. This request URL looks a lot more complicated, with all the parameters at the end:

But everything in square brackets is optional, so you can ignore anything you don’t want to use. Check the Request parameters section of the documentation to see what each of these parameters does.

In our example, let’s use the ‘limit’ parameter to limit our list size to 100, and ‘post_status’ to only return invoices that have been paid. Because these request parameters come at the end of the URL, I can put them into my HTTP action like this:

But it would be exactly the same request if I put them right in the URL like this:

Either way will work, so use whichever is easiest for you.

Request Body

So far we’ve only talked about getting data. But if you want to create or edit data, you need a way to send that data to Blackbaud. This is where the Request body comes in. Post and Patch requests have a Body area where you can send the data you want to create/edit. But you have to send the data exactly how Blackbaud is expecting it, or it won’t work.

SKY API documentation has a Request body area that tells you what information you can send and which elements are required.

In addition to this, they also helpfully include an example request body, so you can see how the data is supposed to be organized. You can copy this schema right into your HTTP action and then replace the default values with your values – either by typing them in, or by using dynamic content.

If you don't need to add one of these properties, you can just delete that line. Just be sure that each property line ends with a comma, except the last one.

There’s lots of room for error here. If you put a comma in the wrong place or use quotation marks where you’re not supposed to, the call will fail. But a failed call doesn’t mess anything up, so you can just look through your request body carefully and try again.

Response schema

So now you have your call set up and when you test it, it’s working perfectly! But when you try to use the data it returned later in your flow, you can’t find it in the dynamic content. This is where the response schema comes in.

Because the HTTP action is so flexible, Power Automate has no idea what it’s going to return, so it can’t add the data to the list of dynamic content like it does for other actions. If you want to use this data later in your flow, you will have to either use expressions, or use a Parse JSON action.

Expressions are beyond the scope of this already quite lengthy post, so I’ll focus on how to use Parse JSON.

Add a Parse JSON action after your HTTP action and set the Content to point to the Body of your HTTP request.

Setting the schema is easy because Blackbaud provides the schema in their documentation. Look for the area called ‘Response 200 OK’ and copy everything in the sample schema box. I chose the Invoice 1099 amount endpoint because the response schema is simple – most endpoints return more data than this.

Once you have it copied, go back to your flow and click ‘Generate from sample’, and then paste the schema in there.

Now you can access the data in the dynamic content list. You’re done!

Heather McLean

@Ben Regier - care to share this at the upcoming User Group? Walk through some of the finer details and tips?

Ben Regier

@Heather McLean
Wish I could, but I'm helping conduct a training at that time and won't be able to make it.

Heather McLean

@Ben Regier - maybe February then?

Ben Regier

@Heather McLean
Actually, I was converting time zones in the wrong direction… I can make it to the January group after all!

Mark Warner

@Ben Regier Thank you for this documentation Unfortunately this didn't work for me. Is this how you correctly create the HTTP request! Or, does this require two separate requests? Mark

Ben Regier

@Mark Warner
All those things in square brackets are there to let you know that you can add optional query parameters if you want to. They aren't actually meant to be part of the URL proper. I see you've already added your query parameters down below, so you can use this url in the Relative path field:

/accountspayable/v1/invoices

Alex Wong

@Ben Regier
This is a great guide. I have been replacing all my normal HTTP action with Blackbaud's HTTP action and here's a few things to note:

• having the query parameter directly in the Relative path does not work for me, getting 404 that the Resource is not found. Only works when I specify the query parameter in the Query section. If this is not the intention of Blackbaud on this connector action, please fix it.

• B/c of the query parameter problem, when getting a list of records (constituent, gift, etc), I need the various parameter in the next_link output, so I uses a few variables to do this
• Variable to hold the full link (initial and subsequent next_link)

• Variable to “create" the query parameter in JSON for each link
  

This allows me to specify the initial url in the Next Link variable, and simply copy these actions and will work for any list that requires multiple Get with next_link.

Glen Hutson

@Alex Wong
Having the query parameter in the Query section is working as designed. In regards to your examples, were you posting that purely as an example or are there things lacking in the main constituent connector that already covers constituent listings, next link, etc?

Alex Wong

@Glen Hutson
I'm just pointing out that having query parameter in the relative path URL field does not work for me, which Ben's original post said you can do it either way. So I am just wondering if some endpoints were allowed (url query parameter specific in the relative path field).

As for your question regarding the purpose of my post, it is not that the Constituent List endpoint has error or something lacking, it is to provide info on how I handle multiple subsequent calls to the endpoint to obtain the complete list of constituent. Each call provide next_link back:

for example:

This next_link need to then be transformed into the relative path and query parameters. My post is my take on how I took the next_link URL and transform it into subsequent calls using Blackbaud HTTP action. There may be others who uses Blackbaud HTTP action that has a different take on how to do this that I would love to hear about.

Glen Hutson

@Alex Wong
Ah, I see what your'e saying. I believe the intent is that query parameters should be in the specific Query Parameters part of the Send HTTP Request. Anytime you may find it working in the URL may be an exception to it. I would plan to always put it in the Query Parameters.

My other question was is there any reason why you're using the Send HTTP Request action to get a list of constituents rather than the version already built into the main Constituent connector? Are there use cases that need to be accounted for/reviewed?

Alex Wong

@Glen Hutson
I am using Blackbaud HTTP for getting constituent list instead of the Blackbaud Constituent connector b/c of data warehousing purpose. I want to get the whole list of 400K+ constituents.

Let me turn my question around to you, or other Blackbaud folks: how would you have done the same. To obtain all 400K+ constituents, I need to make ~92 calls of 5000 (limit on per call). Then uses the next_link provided back on each call. The list constituent action in Power Automate doesn't allow me to provide it a “next_link” to get my next set of 5000. While I can do limit + offset, this does not guarantee the constituent records returned are all unique records. From the Consittuent list API documentation page:

add the sort_token parameter to the next_link URL to ensure that constituents are stably sorted and that the order persists even if changes occur while working through a paginated list

so for data warehouse purpose, using offset and limit doesn't work well while constituent data is add/change/removed. I would love to use offset/limit if what I read/understood on documentation is wrong. Using offset/limit, I can run multiple “thread” concurrently and reduce the time needed to run all 92 loops (1 loop at a time my 92 loop sync takes ~1.5hr, if I can run with concurrency of 3, would reduce my time to ~30 minutes).

Alex

Ben Lambert

Hi all, just wanted to add some clarity as I've seen a few folks get tripped up by this...

The Send an HTTP request action expects the relative path to be the actual relative path, including “path parameters” but excluding “query parameters”.

Path parameters are values specified in discrete segments within the URL, for example /constituent/v1/constituents/280 and /event/v1/events/12345/eventfees (where 280 and 12345 are path parameters representing the constituent and event system record identifiers, respectively).

Query parameters, on the other hand, are values specified AFTER the “?” within a URL. These should be specified separately on the connector action via the “Query parameters” dictionary (key/value pair) field.

In API documentation, both path and query parameters are commonly denoted using curly braces, like {constituentid} and {event_id} but also {include_inactive}. Just know that these parameters need to be specified differently on the connector action in your flow.

At runtime, the connector will compose the URL of the SKY API endpoint to call by building the URL from the SKY API base path (api.sky.blackbaud.com) and the relative path specified on the action (including path parameters). It will then append any query string parameters.

Ben Regier

@Alex Wong
Yes, a big advantage of the HTTP request action that I haven't heard many people talk about is the ability to use sort tokens for syncing data sources.

To be honest, I gave up on using Power Automate to sync RE data to my data warehouse years ago because it was not possible to use sort tokens, which made it more or less impossible to keep all the data up to date.

I use python scripts for this instead, and I will say that they have been rock solid and are so, so much faster than Power Automate for this kind of task.

But as you've demonstrated, if you want to keep your whole process in Power Automate (which I understand the benefit of) you should be able to use sort tokens with the HTTP requests now. Do a little string manipulation to grab that sort token from the next_link, and in your next HTTP request add the sort_token query parameter with that value.

Ben Lambert

Some more thoughts…

Given the complexities with pagination in Power Automate (and Power Apps), we chose (after much internal debate) to omit the next_link property from response definitions in the connector, even though that value is returned by the underlying SKY APIs called by the connector. At the time, we felt it would be better to start simple and evolve in complexity as needed (and based on real-world customer feedback). Including those more-complex properties/parameters also brought challenges around how to properly describe them to users (remember, connectors are more customer-facing than developer-facing, so the bar is high for maintaining the overall user experience).

Given your feedback, I think it's certainly an option for Blackbaud to update the connectors to include next_link (and the sort_token query parameter). So please keep the feedback coming!

Note that it's possible to reference properties returned by an API that aren't listed in the connector definition using an expression. It's a bit more complex, but you can get the next_link value using this expression:

body('List_constituents')?['next_link']

You can see an example of this below:

While more complex, that does work (at least for API responses). Unfortunately, I don't know of a way in Power Automate for a user to pass in additional query parameters that aren't part of the connector definition. So I can't see a way to provide sort_token as an additional parameter made to subsequent calls using the ListConstituents action.

So that's where the Send an HTTP request action can help (since it's more of a developer-facing action than a customer-facing action) where you're manually composing the HTTP request to send.

I'll make sure the team sees this feedback, and they may be able to prioritize adding these pagination fields to the next round of connector updates.

(btw - I've also investigated whether Blackbaud, as the connector publisher, could implement Pagination in the connector itself, similar to what you see on the Settings page for the SharePoint and Excel connectors like you see here. Unfortunately, I don't believe Microsoft supports this for ISV-built connectors, only internal ones. But something to keep an eye on for a future enhancement)

I hope this helps!

Alex Wong

@Ben Lambert
Thank you for your respond. It does seems like the connector isn't gear toward full records dump rather than get a subset based on query filters and provide that quantity of records back (limit being 5000 make sense for purpose you mentioned).

getting the next_link info from the connector actions is not hard, the difficulty has been how to use this next_link field (which is a full URL of next API call to get next 5000 records). Without the new Blackbaud HTTP action, I had to resort to using regular HTTP action which makes handling of OAuth token difficult. But now, with the Blackbaud HTTP action, this has become much easier. Thank you BB.

If you are able to get developer to look into this thread, then I would like to get eyes and respond on this situation I just found out recently. I found it to be a problem so I also reported a Blackbaud support case too, waiting for answer there.

I recently changed the time of night that I do a full constituent dump (I do iterative update every 4 hours throughout the day, but to ensure all my data is correct, I do a full constituent dump at 1AM in the morning previuosly. However, since I moved my full constituent dump to 12AM, I have been getting error on my LAST Loop. Error is due to PK is violated (I dump to Azure SQL Server data warehouse, which Constituent record_id as Primary Key). After some investigation, it appears that when I do my full dump of 400K+ constituent record at 12AM EST, and it takes ~30 minutes, with 92 loops of 5000 limit. My last loop (loop 92) always return some “duplicate” records that was returned in earlier loops. That causes a PK violation. After looking at the data closely, I notice that these records had the “age” property changed all at around ~12:15AM EST. So basically these records were returned once in earlier loop, and DURING my power automate run, BB has some background service that updated the “age” field of donor that has birthday and causes these constituent record to be returned again on the last loop. (by the way, I used date_modified=1970-01-01 to make use of sort_token, which I had assume should have avoided this problem)

So here's my big question, what does sort_token exactly do? for my purpose, dump all constituent records and not get duplicate records or missing records in multiple subsequent calls, what is Blackbaud recommendation?

Thank you

Ben Regier

@Alex Wong

Here's one way to think about the sort_token. When you make a call to the Constituent List endpoint and set last_modified to 1970-01-01, you're basically asking for a list of all constituents that have been modified since that date, right?

The sort_token provides you with a way to ensure that all modified constituents come to you in the proper order. Even constituents that are modified after you made your first call.

The problem you are seeing is happening because there are apparently some constituents that are being marked as modified after you start your sync process. So they're being fed to you twice, which is causing your PK errors.

(a quick note here that there are a number of things that happen behind the scenes that can cause a record to be “modified” in the API - so this doesn't actually mean that people are modifying these records in the middle of the night)

The downside of this behavior is that if you want to use sort_tokens properly, you're going to have to check if a record exists in your data warehouse so you'll know whether to Update or Insert it. (Or just use run-after configuration to do an update if the insert fails.)

The upside to this is that if you store your sort token somewhere, you can use it anytime in the future and continue syncing right from where you left off without missing anything. You could set it to sync every minute if you wanted to, and it would only pull in the records that were modified since your last sync.

I have found this method to be so reliable that I have not had to rebuild my data warehouse since I started using sort tokens. I only refresh the constituents that have been modified.

Alex Wong

@Ben Regier
Thank you, this is very helpful. I will make better use of the sort_token and check for PK violation situation.

Alex Wong

@Ben Regier
I just remember why I have a full dump at night and an iterative update during the day, every 4 hours: it is because of constituent that gets deleted. However, now that I have a webhook on the constituent delete event, I can delete the constituent from the data warehouse. So I think I do not need to have a full dump of constituent nightly any more.

However, I don't think I can get rid of my gift nightly full dump. Considering constituent merge (which causes the constitent gift to “move” to another constituent record) or the webview function to move a gift to another constituent record. This “change” does not cause a gift record to be consider “last changed” and therefore not show up in an iterative list of record last changed in last XX hours. This can cause data warehouse error on the gift table's constituent_id field, soft credit table's consittuent_id field, and fundraiser table's constituent_id field. From your perspective, is there any ways to avoid doing a nightly full gift tables dump?

Ben Regier

@Alex Wong
We don't sync the gift table, so I might not be much help there.

You're saying that if you move a gift from one constituent to another, it doesn't change the last_modified date? Yes, I could see how that could be difficult to work around. Have you tested the Gift Changed webhook to see if it catches those sorts of events?

Alex Wong

@Ben Regier
No I did not test if gift change webhook fire, since when I change the gift to another constituent, the gift property of last change did not update (webview and database view). I assume that the gift change webhook would not fire. (as there is really not a direct “change” to the gift information.).

I will give this a test and report back

Alex Wong

@Ben Regier
First of all, I think the discussion has gone side way from the OP. Maybe should move to another discussion thread, but is it something BB can do?

So I tested the situation with webview “moving” of gift from one constituent to another. This DOES trigger a gift change and it DOES show up when using GET Gift List with last modified date. So this is promising.

However, I went ahead and tested a few other situation:

• Added a soft credit on the gift in database view - GOOD
  • does trigger gift change webhook and does show up in GET Gift List with last modified date
  • DOES NOT cause database view's gift property of last changed (this is very odd)
• Merge (using webview) the record that has the soft credit to another record - BAD
  • does NOT trigger gift change webhook and does NOT show up in GET Gift List with last modified date
  • I do know webhook sometimes take a while to trigger, so for time reference, I did the merge ~7:30AM this morning. 2 hours later, 9:30AM, no webhook trigger, not show up in GET Gift List with last modified

For this reason, my gift soft credit table can be mess up with wrong / non-existing constituent_id field. So what I do currently is to run a FULL gift dump nightly while only do iterative during the day.

Any other ideas?

Ben Regier

@Alex Wong
Yes, that is a tough one. I've sometimes seen webhooks take as long as 12 hours to deliver an event - any chance that the merged gifts showed up eventually?

If not, one thought I had was to use your Constituent Delete webhook to check for orphaned soft credits. I'm assuming here than when two constituents are merged, one of them is deleted and that this will trigger the Constituent Delete webhook.

If so, you could include some logic that would check your soft credit table in your data warehouse for any gifts credited to the constituent ID that was just deleted. If they are found, you could use the gift ID to grab the new constituent ID from Blackbaud and then update your gift record in your data warehouse?

This is just thinking out loud - I haven't tested any of this.

Alex Wong

@Ben Regier
No, did not show up. I waited 8 hours before I deleted the test gift. (when I delete the test gift, the delete webhook kicked off).

What you said as a possible solution I gave it some thought too, but the issue is RE NXT webview merge does not have option to delete the record, just mark as inactive, to delete, one will need go to into database view. Because of that, my CRM team does not delete merged record right away, instead they usual delete these records once a week. So from that, even if I have “handling” in the constituent delete trigger, my data warehouse would be incorrect for days.

While I can tell my crm team to make sure to delete the constituent record right away (they will be reluctent as database view has been slow performing).

I think a possible solution is, I can create a SKY Addin action button on constituent to delete the constituent by triggering a flow that deletes the constituent, plus any other logic to handle any database table/field that is affected by this delete (this avoids the delay that webhook may have). I can also create an adaptive card on the homepage that allow them to specify a list of constituent id to be deleted that does the same. I'm sure this way they will be more willing to delete right away.

What do you think of this approach?

Thanks, sometime “talking” it out helps comes to a solution. Love that about community

Ben Regier

@Alex Wong
Yes, building your own delete constituent functionality into the web view seems like a good option - gives you control over the process and your users are likely to use it since it's easier than opening Database view.

Of course, it would give all of your users delete permissions unless you include some logic in your flow locking it down to specific people, but you've probably thought of that.

I'm still surprised that merging a record doesn't trigger any of the mechanisms that mark a gift as modified. That doesn't seem like it would be the intended behavior. Might be worth putting in a ticket with Blackbaud to make sure they're aware of that.

Alex Wong

@Ben Regier
Yes, I added logic to check a SharePoint List where I kept all “Add-in” functionalities that is authorized for certain people and the flow check for that before allowing.

I will open a ticket to have it investigated, but given my last few documentation error support ticket's experience, it's going to be a bit tough to get through first level support to have it submitted and looked at by an engineer.

I can't find it anywhere, so if it doesn't exist, I like to know if Blackbaud can put together a documentation that specifically talk about what will cause record to be consider changed.

For example, constituent custom field change does not mean constituent record change. but FE project custom field change DOES mean project record change. I feel a little more docuemtnation or information about it will help those that need to do data warehousing without the need to test out every case before saying a full dump is needed.

Austen Brown

@Ben Regier - Thank you so much for posting this and including screenshots. Very helpful!

Hallie Guiseppe

@Ben Regier

Good morning!

I am working on a flow that will email a list of event participants the week of an event. I have two issues at this stage.

First, thanks to your post I am using the "Send an HTTP request to get a list of events. I'd like to filter on event date but it doesn't seem I can do that with the Query parameters. I tried startdate > 6/2/2023 and enddte is >6/8/2023 but that doesn't seem to do anything. I also tried using startdate = UTCNow() and enddate = adddays(UTCNow(),8) but that also didn't work. I decided to try and finish out the flow and work on the date range later so I limited the flow to 5 events. That is working well. I get all the event info and the flow runs successfully to this point.

My second issue occurs when I try to parse the json. See error and schema below. I believe I followed your instructions correctly generating a Schema from the sample provided on the endpoint reference page.

Schema generated from BB sample:

Error I get when testing: “The schema validation failed.”

I can't move on to the next step until I isolate the event id for “list event participants” action.

Thanks for your help!

Hallie

Alex Wong

@Hallie Guiseppe
first of all, regarding your event date range, you can try in console first to make sure you getting the “date string” correctly. This works ok for me and only return event that is in June 2023:

stick with yyyy-mm-dd format.

your screenshot is very hard to see, so if you want to paste your JSON Schema as text that will help.

Hallie Guiseppe

@Alex Wong

here is the json schema that generated from the sample:

{

"type": "object",

"properties": {

"count": {

"type": "integer"

},

"value": {

"type": "array",

"items": {

"type": "object",

"properties": {

"id": {

"type": "string"

},

"lookup_id": {

"type": "string"

},

"name": {

"type": "string"

},

"start_date": {

"type": "string"

},

"start_time": {

"type": "string"

},

"end_date": {

"type": "string"

},

"end_time": {

"type": "string"

},

"attending_count": {

"type": "integer"

},

"invited_count": {

"type": "integer"

},

"revenue": {

"type": "integer"

},

"goal": {

"type": "integer"

},

"percent_of_goal": {

"type": "integer"

},

"date_added": {

"type": "string"

},

"date_modified": {

"type": "string"

},

"capacity": {

"type": "integer"

},

"inactive": {

"type": "boolean"

},

"attended_count": {

"type": "integer"

},

"category": {

"type": "object",

"properties": {

"id": {

"type": "string"

},

"name": {

"type": "string"

},

"inactive": {

"type": "boolean"

}

}

}

},

"required": [

"id",

"lookup_id",

"name",

"start_date",

"start_time",

"end_date",

"end_time",

"attending_count",

"invited_count",

"revenue",

"goal",

"percent_of_goal",

"date_added",

"date_modified",

"capacity",

"inactive",

"attended_count",

"category"

]

}

}

}

}

Alex Wong

@Hallie Guiseppe
Use this instead:

3 issues you have when you generate from the sample: (1) integer and number type is not always right, you want to check the Schema section to make sure. (2) the “Required” section is generated, which I always delete. (3) some properties may not be used in your RE NXT, so it will come back as null, so if you still have issue, you can do this instead for each properties:

"id": {},
"lookup_id": {},

{
"type": "object",
"properties": {
"count": {
"type": "integer"
},
"value": {
"type": "array",
"items": {
"type": "object",
"properties": {
"id": {
"type": "string"
},
"lookup_id": {
"type": "string"
},
"name": {
"type": "string"
},
"start_date": {
"type": "string"
},
"start_time": {
"type": "string"
},
"end_date": {
"type": "string"
},
"end_time": {
"type": "string"
},
"attending_count": {
"type": "integer"
},
"invited_count": {
"type": "integer"
},
"revenue": {
"type": "number"
},
"goal": {
"type": "number"
},
"percent_of_goal": {
"type": "integer"
},
"date_added": {
"type": "string"
},
"date_modified": {
"type": "string"
},
"capacity": {
"type": "integer"
},
"inactive": {
"type": "boolean"
},
"attended_count": {
"type": "integer"
},
"category": {
"type": "object",
"properties": {
"id": {
"type": "string"
},
"name": {
"type": "string"
},
"inactive": {
"type": "boolean"
}
}
}
}
}
}
}
}