Automate CSV to Parquet with Hugging Face Buckets, Webhooks & Jobs

Author: Hugging Face · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Automate CSV to Parquet with Hugging Face Buckets, Webhooks & Jobs · Channel: Hugging Face · 04:50 · Watch on YouTube

Topics: Task and workflow automation, Data, databases and embeddings, AI infrastructure and hardware

Summary

The video demonstrates how to automate converting CSV files to Parquet using Hugging Face Buckets, webhooks, and jobs. It sets up an input bucket and a separate output bucket, then creates a job to process CSV files. A webhook watches the input bucket and triggers the job when a new file arrives. The demo syncs a test CSV to the input bucket, checks that the job completes, and confirms the Parquet output in the destination bucket.

Key points

Buckets are useful for files that arrive over time, such as logs and new data files.
The workflow uses separate input and output buckets.
The job uses a script called optimize-parquet.py to convert CSV files into optimized Parquet files.
A job can be created before a webhook, but without a webhook ID the script does not process a file.
The webhook listens for changes in a specified Hugging Face repository or bucket and uses the job ID to trigger processing.
The demo syncs trip-data-new.csv to the input bucket using hf buckets sync.
The job completes successfully, and its logs show the output location.
The resulting Parquet file appears in the output bucket.

Chapters

Start	Chapter	Summary
00:00	Overview of the workflow	The video introduces a workflow that uses buckets, webhooks, and jobs to convert newly uploaded CSV files into Parquet.
00:30	Set up the buckets	The demo selects input and output buckets and explains how to list and create buckets, including making one private.
01:35	Create the processing job and webhook	A job is configured to run the conversion script, then its ID is used to create a webhook watching the input bucket.
03:13	Upload a test CSV and check the job	A test CSV is synced to the input bucket, triggering the webhook and job, and the job status and logs are checked.
04:19	Verify the Parquet output	The video lists the output bucket recursively to confirm that the newly created Parquet file is there.

Quotes

We’re going to take a new CSV file, put it into a bucket, watch a webhook, pick it up, give it to a job, and create a Parquet file. — 00:00

The best part is, it can all be done on the cloud, without having to download and upload these large files back and forth. — 00:30

We need to create a job that the webhook can call when a new file appears in the bucket. — 01:35

And now that’s there, now we should be able to see how this is running with our webhook and job. — 03:48

Concepts and entities

Hugging Face, Buckets, Webhooks, Jobs, Xet, CSV, Parquet, optimize-parquet.py

Full transcript (original language)

00:00 Hey everyone! In today’s Quick Hug, what we’re going to do is use webhooks, buckets, and jobs. We’re going to take a new CSV file, put it into a bucket, watch a webhook, pick it up, give it to a job, and create a Parquet file. Let’s get going. Buckets are good for files that arrive over time, things like logs, scans, new data files. Once you have these data files in the bucket, you’ll want to convert them, clean them, and do data engineering tasks. Webhooks and jobs can automate all this for you.

00:30 The best part is, it can all be done on the cloud, without having to download and upload these large files back and forth. We’re going to need two buckets for this to work. The first one is where the files initially end up inside of a Hugging Face storage bucket. As we process them, we need a second bucket for those processed files to land into. Let’s have a look at the buckets that I currently have. To see what buckets you already have associated with your account, you do hf buckets

01:01 ls. If I do that, you’ll see I have three buckets. For this demo, we’re going to use NYC taxi data as our input bucket and NYC taxi parquet as our output bucket. Remember, we’re converting a CSV file into a parquet file. To create a bucket is super easy. What you do is hf buckets create the location of the bucket. Of course, put your username in front of it. And just to show you how I created the parquet bucket. We did it like this, and then I added the private flag so that it’s only private to

01:35 me. If you leave that private flag out, the bucket will be public. We need to create a job that the webhook can call when a new file appears in the bucket. What I have here on the screen is the definition of a job that uses this specific script, which is optimize-parquet.py, to go and take a CSV file and create an optimized Parquet file with it. When you typically run a job on Hugging Face Hub, it will run once and do whatever the script is supposed to do.

02:07 In this case, this specific script looks for a webhook ID so that it will go do some process. But because we haven’t created a webhook yet, it doesn’t actually do anything when we run this command. Now let’s talk about webhooks. A webhook allows you to listen for changes on a specific repository or bucket that belongs to a specific user or organization—not just your own, but anyone. That’s on Hugging Face. When we ran the job, it gave us a job ID and it actually did run, but there’s no

02:40 webhook, so nothing actually happened. But what we need is this job ID right here. So we’re going to take that and we’re going to copy it and use it in the next step to create the webhook. Okay, so what I’m going to do is put the job ID into an environment variable that we’re going to use in creating the webhook. So there’s a job ID and here’s the command for my webhook. I say hf webhooks create. There’s a job ID. Watch this bucket again. This is our input bucket NYC taxi data, and then I say use the current logged

03:13 in user’s Hugging Face token. Now if I go run that, that will say voila, we have a webhook now created. All right, so let’s create a new test file inside of our bucket. I’m going to just grab the first ten thousand rows of our original data file and create a new file called trip-data-new.csv. Oh, that will go and create that. You’ll see here’s our file. Now let’s sync that. Using hf buckets sync, this directory, hf colon slash slash buckets my username and then the actual bucket NYC taxi data.

03:48 Now this will run pretty quickly because it’s using Xet underneath the hood to do deduplication. And now that’s there, now we should be able to see how this is running with our webhook and job. Okay, our job is finished running. Let’s go check on its status. We’ll do hf jobs ps -a, and then we’ll see that the job that we just ran finished with the status of completed, and it took about eight seconds. Now we can go look at the output of that job if we wanted to. We can do hf jobs

04:19 logs, and then put in the ID of that specific job, and then you’ll see that here is the output, and the message was Wrote buckets/prpatel/nyc-taxi-parquet/trip-data-new.csv. Now, if we wanted to, we could also go and look to see what is actually in that bucket. So we could do hf buckets ls prpatel/nyc-taxi-parquet.

04:50 And again, this is the output bucket that we were writing to. So let’s go run that, and we’ll put the -R on it to do recursive. And here we are. Here’s our trip-data-new and our Parquet file that was just created. All right. So that’s it. I hope you enjoyed that video. If you look in the links, I have a link to my repository where all my notes and demo code are, and I’ll also include a link to the full tutorial on using webhooks, so that you can start experimenting with this on your own.