About Pachyderm

Pachyderm is a data foundation for ML / AI. The software helps provide users with the needed data layer that allows MLOps and data science teams to scale and produce throughout the machine-learning lifecycle. The software offers users tools such as petabyte scalability, data driven automation and end-to-end reproducibility. The platform is user-friendly and helps teams with their ML projects by lowering their data processing and storage costs, marketing them more efficiently and meeting all regulatory compliance requirements.


Read More

Supported Operating System(s):

7 Reviews of Pachyderm

Average User Ratings

Overall

4 / 5 stars

Ease-of-use

3.5

Value for money

4.0

Customer support

5.0

Functionality

4.5

Ratings Snapshot

5 stars

(1)

1

4 stars

(5)

5

3 stars

(1)

1

2 stars

(0)

0

1 stars

(0)

0

Likelihood to Recommend

Not likely

Very likely

Filter reviews by:

Sort by: Most Helpful | Most Recent

Showing 1 - 7 of 7 results

Do you use this product?Write a review

October 2021

Chris from Prescient

Company Size: 2-10 employees

Industry: Marketing and Advertising

Time Used: Less than 12 months

Review Source: Capterra


Ease-of-use

3.0

Value for money

4.0

Customer support

5.0

Functionality

5.0

October 2021

Scalable machine learning without the mlops

Pros

The ability to scale model builds in native python is something that has been missing in this space until now. Utilizing spark and/or dask comes with a large amount of overhead that can be avoided leveraging pachyderm.

Cons

The learning curve is quite steep since there are some core concepts that are foundational to understand before using pachyderm.

Response from Pachyderm

Replied November 2021

Thank you for your review Chris!

November 2021

Clayton from Anthem AI

Verified Reviewer

Company Size: 10,000+ employees

Industry: Hospital & Health Care

Time Used: Less than 2 years

Review Source: Capterra


Ease-of-use

3.0

Value for money

4.0

Customer support

5.0

Functionality

5.0

November 2021

Rethinking Data in AI and ML

Like any tool, Pachyderm is no silver bullet for the entire AI/ML stack. However, from a data processing and management perspective, it has fulfilled every application requirement I've needed it for and continues to be a flexible tool in meeting additional requirements. For example, after having computed some results from a pipeline, I needed to serve these results to an existing application. Pachyderm made this simple by exposing the data through a built-in S3 REST API. Since the application was already compatible with S3, Pachyderm served as a drop-in replacement for an S3 bucket. For anyone that strives to design clean and straightforward AI/ML architectures, I can definitely recommend Pachyderm as a must for the foundational data component.

Pros

AI/ML production systems typically consist of multiple data processing steps organized as a DAG. Many automation frameworks manage these DAGs as tightly coupled steps ordered by _code execution_. What I like so much about Pachyderm is that it approaches DAG management as loosely coupled steps ordered by _data dependencies_. This alternative way of thinking has enabled me to design AI/ML architectures with data at the center, which has revolutionized the development and production workflows I've participated in. I can confidently store, process, and otherwise manage the data because Pachyderm provides a solid foundation for data provenance, data versioning, data storage patterns, and efficient incremental processing. Since AI/ML models are effectively a form of data, model versioning and management can be built as an extension of Pachyderm's data foundation. Furthermore, I really like that Pachyderm is powered by Kubernetes, because it passes on important architectural properties to Pachyderm, such as high scalability, robustness, efficiency, and portability (i.e. cloud agnosticism). I can containerize my pipelines, quickly test them locally through Docker Desktop or minikube, then scale them up to massive amounts of data in an on-prem or cloud cluster. If autoscaling is supported in a cloud cluster, I can especially reap the benefits of cost efficiency because I only pay for the compute resources I use.

Cons

- In 1.X versions of Pachyderm, there are a few performance pain points, especially around handling very small files when uploading/downloading to/from a repo. These pain points have been significantly improved in Pachyderm 2.X. - Also in 1.X, debugging pipeline failures can sometimes be challenging without extra tools or integrating external logging services. Pachyderm 2.X improves upon this as well. - When Pachyderm processes data files in a pipeline, it groups the files into logical structures called datums for provenance and data efficiency reasons, and then it invokes the pipeline on each datum. This is necessary for scalability, but the downside is that each invocation of the pipeline incurs an overhead cost of just starting the processing code. The bright side is that there are several straightforward ways to engineer around the problem. It's also important to recognize that the impact of the problem is minimized by the benefits of incremental processing(i.e. only processing data that has changed on future pipeline runs). - This isn't necessarily a problem, but prospective buyers should be aware that although compute costs may go down due to incremental processing, storage costs may go up due to storing multiple versions of data.

Reasons for Choosing Pachyderm

Although DVC provides data version control features and AI/ML pipeline management, it lacks containerized pipeline orchestration and seems better suited for small teams in startup or research environments. We needed an enterprise-level service.

Reasons for Switching to Pachyderm

Airflow is mainly geared for pipeline orchestration. My team had to build in a custom data management layer, but there was much to be desired in terms of provenance and versioning. Since Pachyderm already provided these features plus pipeline orchestration, it made more sense to not reinvent the wheel with Airflow.

Response from Pachyderm

Replied November 2021

Thank you for your very thorough review Clayton.

October 2021

Martin from Karius

Company Size: 51-200 employees

Industry: Biotechnology

Time Used: Less than 12 months

Review Source: Capterra


Ease-of-use

2.0

Customer support

4.0

Functionality

4.0

October 2021

Great in theory

We achieved some of our goals with Pachyderm. However, we were really hoping to spend more time on solving the problems directly related with our goal. Instead, we spent a significant amount on time solving problems with Pachyderm and tailoring our problem to it.

Pros

Great concept, really fits what we would like to do. Re-computing only the pieces where the data has changed is super valuable.

Cons

Working with it in practice is very hard. We would like to use Pachyderm also for research, developing research pipelines that can be executed easily on big amounts of data on the cluster. However, during research/development, pipelines naturally crash often. Translating something that works locally to something that works in pachyderm has several scenarios in which it can fail. Inspecting those types of errors is incredibly difficult, unless you invest a significant amount of time into setting up logging/monitoring manually.

Response from Pachyderm

Replied November 2021

Hello Martin, thank you for your feedback, we truly appreciated it. Pachyderm 2 will have several enhancements around the troubleshooting workflow for pipelines and the new Console (dashboard) will likely be of great help here. However, we're striving to further improve the user experience of Pachyderm with every release. Thank you.

November 2021

Cove from National Ecological Observatory Network

Company Size: 201-500 employees

Industry: Research

Time Used: More than 2 years

Review Source: Capterra


Ease-of-use

4.0

Customer support

5.0

Functionality

5.0

November 2021

Game changer for handling dynamic data

Pachyderm meets many previously unmet needs for our organization, including complete data provenance, automatic handling of data change, and modular/portable processing architecture, which facilitates the joint development of processing pipelines between software developers and scientists. Pachyderm engineers have been extremely responsive to our issues and development requests, and we plan to work well into the future with this software.

Pros

Perhaps the most important aspect we benefit from operationally is the awareness and automatic handling of data change. Generation of our data products involves multiple processing steps and several sources of data and metadata that enter the processing sequence at various points and may change at any time. Pachyderm automatically knows what has changed and triggers downstream (re)processing, removing the need for error-prone human management.

Cons

In Pachyderm 1.X there was a relatively high amount of overhead associated with processing each datum. Our data typically consists of small but numerous datums, and we needed to artificially combine datums for performance. However, Pachyderm has been working with us on this issue and we expect to see big improvements in 2.0 and beyond.

October 2021

Xubo from Karius, Inc

Company Size: 201-500 employees

Industry: Biotechnology

Time Used: Less than 2 years

Review Source: Capterra


Ease-of-use

5.0

Value for money

4.0

Customer support

5.0

Functionality

5.0

October 2021

Pachyderm is a great data processing platform on cloud.

We have used Pachyderm for more than a year. Overall experience is Good. We love the core technology and features provided by Pachyderm. We experienced frustrated issues, like the download speed, deployment, system stability. We get excellent support from the Pachyderm team all the time.

Pros

Data Driven Automation. It supports incremental data processing. Reproducibility. Perfectly match our tech stacks: K8s, S3. Community facing.

Cons

We expect fully automated data replication/export to external storage system. The logging & debugging support could be improved.

Reasons for Choosing Pachyderm

Data Driven Automation. It supports incremental data processing. Easy integration with our infrastructure.

Response from Pachyderm

Replied October 2021

Xubo, Thank you for your review, we greatly appreciate your feedback. We'll make sure to pass your feedback around logging and debugging on to our product team. - Pachyderm

October 2021

Chris from Riskthinking.AI

Company Size: 2-10 employees

Industry: Information Technology and Services

Time Used: Less than 2 years

Review Source: Capterra


Ease-of-use

4.0

Value for money

4.0

Customer support

5.0

Functionality

5.0

October 2021

Pachyderm for data pipelines

Pros

Pachyderm pipelines are an intuitive way to split and process data concurrently using autoscaling compute clusters. Writing a program to interact with data in a pipeline is straightforward due to working similar to a native filesystem, requiring no additional libraries or integrations.

Cons

We ran into issues with Pachyderm that required deleting and recreating pipelines. As an upside, support was very responsive to resolving our problems and providing upgrades to Pachyderm.

Response from Pachyderm

Replied November 2021

Chris, Thank you for your great feedback. We're glad to hear that our support team has been a great asset to you. We'll make sure to pass along the feedback.

November 2021

Will from Adarga

Company Size: 51-200 employees

Industry: Information Technology and Services

Time Used: Less than 12 months

Review Source: Capterra


Ease-of-use

2.0

Value for money

4.0

Customer support

5.0

Functionality

3.0

November 2021

The missing ingredient for reproducible research

I'm a big fan of the pachyderm approach; it's young software and needs to be understood a little to get the best out of it; but when stuff works, it works so damn well.

Pros

The systematic recording of provenance for training and benchmarking results.

Cons

When things go wrong, it's hard to diagnose.

Reasons for Choosing Pachyderm

For NLP, the requirements around data curation for training are slightly singular, pachyderms offering was the only sensible one for us.

Response from Pachyderm

Replied November 2021

Thank you for the review, Will.