SFTP File transfer over 50MB - anyone try chunking files?

Options
Torie Wright
Torie Wright
Eighth Anniversary Participant Loyalty Badge Blackbaud Champion
in SKY Developer General Discussions

I have a file that I export via queue into hosted files to be pushed to my sharepoint via powerautomate. I just found that the reason this particular file has not been syncing is because the SFTP trigger skips files over 50MB (which this file is). Has anyone been able to implement chunking successfully?

Tagged:

Comments

  • Alex Wong
    Alex Wong Community All-Star
    Ninth Anniversary Kudos 5 PowerUp Challenge #3 Gift Management Name Dropper

    @Torie Wright
    over 50MB of csv file is a bit much.

    I have not done chunking before, as I didn't have this need. Even if chunking can do what you need it to (download >50MB file). I think you still will want to revisit filter (row) and output (column) of the csv and see if you can reduce this file size potentially by remvoing duplicate data row and/or column.

    For example, if you have a gift query that output a whole bunch of constituent data columns (i.e. address, constituent code, constituent attribute, phone, email, etc), any donor with multiple gifts will get “duplicate” data that makes your csv huge. Instead you can opt to have 2 queries: one for constituent that output constituent data and one for gift, for gift data. This obviously also depend on how you plan to use/consume this csv files. Thinking through all of these will help you have a better data extract stretegy.

Categories