Incremental/delta data export using command line data loader
A few months back, we got a requirement for hourly incremental/delta export of data from Salesforce. There was a restriction that we didn't want to purchase any other licensed ETL tool, so we decided to use the command line data loader using that export can be scheduled also. Command line data loader requires SOQL to export the data, so to export the data hourly we scheduled the process using windows scheduler for every hour, but the main challenge here was to write a dynamic SOQL that would apply a filter to extract the records for those created date is greater than last 1 hour. Salesforce does not provide any SOQL datetime literal like LAST_1_Hour or LAST_N_HOUR. There are 2 methods to accomplish the requirement. First is using powershell script and the second is using SOQL itself. Method 1 - PowerShell Script In command line data loader, process-conf.xml contains the SOQL, so in case of an hourly incremental data extract SOQL will be like SELECT Id, Name FROM Contact ...