Load data into Vertica from an HTTP source using cURL – with example

Category : Code · by Jul 27th, 2014

Recently I needed to load a CSV located in a public Amazon S3 bucket into Vertica. There’s some Vertica documentation for setting up a User Defined Load (UDL) Source, but here I’ll show you how to bring it all together.

In short, there’s a UDL Source for cURL in /opt/vertica/sdk/examples. You build the UDL from the given cpp code using make, and then create a library and source in Vertica for cURL. Finally, you can use cURL with a COPY statement to load data into a table.

SHARE :

(2) comments

Tariq
2 years ago · Reply

Hi Sohan,

Thank you for the nice post. I have been trying to build cURL module, but unable to do so. Whenever I run make I get the following error :

c++ -I /opt/vertica/sdk/include -I HelperLibraries -g -Wall -Wno-unused-value -shared -fPIC -I -o /opt/vertica/sdk/examples/build/ScalarFunctions.so ScalarFunctions/*.cpp /opt/vertica/sdk/include/Vertica.cpp
clang: error: no such file or directory: ‘/opt/vertica/sdk/examples/build/ScalarFunctions.so’
make: *** [/opt/vertica/sdk/examples/build/ScalarFunctions.so] Error 1

I would really appreciate some help. Many thanks.

sohan
2 years ago · Reply

Hey Tariq,

Double check that you were able to compile some of the required cpp packages:

# see what packages you have built. You want ScalarFunctions.so to exist here
ls -l /opt/vertica/sdk/examples/build/

# compile some required packages
cd /opt/vertica/sdk/examples
make clean
make

Best,
Sohan

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>