NetApp - Dave's Blog

Three Ten-Year-Trends That Are Driving IT Change

A trick I use to predict the future is to identify ten-year-trends. These are really big trends that take place over a decade or more. At the moment, there are three ten-year-trends that interest me, but before I get into that, let me give some historical examples to clarify what I’m talking about.

In 1981, there were only an elite and nerdy few of us who had computers on our own desks. (I built my first computer in 1977, at age fourteen, but I am … different.) There were more people with some kind of remote terminal, but the majority of business people had no personal access to computers at all. Then, in 1982, IBM introduced the PC, and by 1992, ten years later, pretty much everybody in business had a computer on their desk. The ten-year-trend in IT during that decade was “put a computer in front of everybody’s face.”

During the 1990s, the ten-year-trend was “wire all of those computers into one big network.” At the beginning of the nineties, many computers weren’t networked at all, and networks that did exist were often small, connecting just a handful of computers in nearby offices. By 2000, the Internet was ubiquitous. IBM was running TV ads with nuns on camels in the middle of the desert sending e-mails to each other.

When I see lists of “this year’s top trends,” they often feel small and short-sighted to me. Any trend that only lasts a year is too insignificant to be important. It’s probably mostly over by the time you spot it. The really interesting trends, the ones that drive big change, last many years—a decade or more.

The three ten-year-trends that I see in IT today are: 1) Cloud/Outsourced Computing, 2) Server Virtualization, and 3) Flash Memory.

The big question behind cloud computing (or outsourced computing) is whether a company should build or expand its own data center, or whether it should access computing resources remotely, over the Internet. People are already using lots of cloud/outsourced computing today for things like blogging and web training, but I expect it to move up-market over time. This won’t happen overnight, but ten years from now I believe that many medium-sized businesses and a few large enterprises will have cloud-outsourced pretty much all of their computing infrastructure. (The term “cloud computing” is confusing because so many definitions are floating around. I’ll give mine in an up-coming blog.)

For people who do build data centers (either because they ignore the cloud/outsourcing trend or because they provide cloud computing to others), server virtualization will radically change how those data centers are built. Server virtualization is at the center of the trend, but it pulls a boatload of other trends along in its wake, including blade farms, network virtualization, and (my favorite) storage virtualization. Data centers built in ten years will look radically different than they look today.

Flash memory is amazing stuff. It is the first technology in decades with the potential to create a pervasive new layer in the storage hierarchy. Many technologies over the years, like bubble memory, have been touted as “disk-drive killers,” and none of them ever panned out, but flash memory is emerging as an enormous force, and there is no question that storage systems will look radically different in ten years. I’m not predicting that flash memory will eliminate disks any time soon, certainly not in five years and probably not in ten years, but I do think that it will relegate disks to an increasingly tape-like roll as flash absorbs more and more of the I/O intensive loads that disks handle today.

These are the three ten-year-trends that I see driving change in IT, but I’d love to hear from readers if they think I’ve missed some, or that the ones I’ve chosen won’t drive as much change as I think they will.

Disk Is The New Tape—Flash Is The New Disk

Almost every measure of computer performance increases exponentially. Here is one important exception: disk drives keep getting bigger, but they are not getting much faster. As a result, the number of seeks-per-second available for each gigabyte of data (seeks/second/GB) is plummeting.

Let me give a concrete example. Fifteen years ago, a typical one-gigabyte disk drive had a seek time of ten-milliseconds or so. Do that math, and that’s 100 seeks-per-second for that one gigabyte. Today, one-terabyte drives have maybe five-millisecond seek time. Do the math, and that’s 200 seeks-per-second for the whole terabyte, or just 0.2 seeks/second per gigabyte. Over the past fifteen years, your ability to access the data you own has gone down by a factor of 500.

That’s bad enough, but if you consider how many CPU instructions you can execute while you wait for the disk drive, things are even worse. Fifteen years ago, the Intel 486DX could do 54 MIPS (million instructions per second). Today Intel’s QX9770 can do 59,455 MIPS. For every millisecond you wait, today’s chips execute 1000 times more instructions.

Consider these facts together: From a human perspective, seeks/second/GB has gone down by a factor of a five-hundred, but from the CPU’s perspective, it is five-hundred-thousand times slower! Remember those old mainframe computers with tape drives on the front, jerking back and forth? To the CPU, tape drives back then seemed faster than disk drives today.

Disks are the new tape—worse than tape used to be—so we obviously need to find a new disk. That’s where flash comes in: ten times more expensive than disk, but for random access, a hundred times faster. For now, flash is mostly used in small portable devices, or for very high performance, but if you look forward five or ten years, I predict that people will think of flash the way they think of disks today—as the fast-access storage for everyday use. And they’ll think of disks as being more like tape—a slow-access media that’s useful for stuff you might want to look at someday, but not particularly useful for data you use all the time.

Flash is too expensive to replace disk right away, so first we’ll see a new generation of storage systems that combine the two: flash for performance and disk for capacity. I hesitate to compare this with the old HSM (hierarchical storage management) solutions that combined disk with tape, because those were so sucky—things that would take seconds on disk could sometimes take hours. They only worked for a limited subset of applications, and only with intricate and painful management. Fortunately, the performance ratio between flash and disk is much better than for disk and tape, so we will be able to automate the management and still get good performance. HSM without the M, if you will.

(This article is mostly about the industry in general. For more on what NetApp is doing, check here and here.)

Server Virtualization Trend: Just Starting or Almost Done?

Something has been bugging me about the market share numbers for server virtualization. Is the trend is just getting started, or is it almost finished? The numbers I’ve seen say that under 10% of all X86 servers have been virtualized – maybe 7-8%. By that measure, the trend of converting physical servers into virtual ones seems to be quite early.

Things look very different when I look at the percentage of total servers (physical plus virtual) that are virtual. Most customers seem to run at least 8-12 virtual servers per physical, and some are pushing past 30 towards 50. Let’s use 10 as a conservative number, and do the math: For every 100 physical servers, 7 are virtualized, for a total of 70 virtual servers. That makes a total of 163 total servers (70 + 93), and almost half are virtual. If we are half-way converted, then the virtualization trend must be very far along, because the second half will probably convert much faster than the first half.

It sure seems to me that looking at total servers is the right thing to do, as opposed to just counting how many physical servers are virtualized, because to a user, it shouldn’t make any difference whether their server is virtual of physical. (That’s the whole point!)

On the other hand, it doesn’t feel right that server virtualization is so far along. Most customers I talk with are just getting started. Only a few have seriously converted. My math must be busted, because there’s no way that we are half-way converted.

I think the problem is that the math assumes that there is a fixed-size pool of servers that people are converting from physical to virtual. It seems more likely that cheap and easy-to-provision virtual servers will lead to a massive increase in the total number of servers. That is always what has happened when a computing resource gets much less expensive. We didn’t just replace workstations with PCs, we gave PCs to everyone, instead of just development engineers. Likewise, the cost per gigabyte keeps dropping every year, but instead of buying fewer gigs, people keep storing more and more, and their budgets stay roughly flat.

Given the history of the computer industry, it seems unlikely that server virtualization will drive costs down in the way people think. Instead, it seems much more likely that costs will stay roughly flat, but there will be a radical proliferation in the number of virtual servers. They are just so fast, easy and cheap to deploy, it seems likely that most IT shops will hand out scads of them.

I’m not sure whether to think of this as a prediction, or a warning. I guess if you get value from all those virtual servers (just like we did from mini-computers, workstations, and then PCs), then there’s no problem. But if IT shops really want to use server virtualization to save money, then they had better be extraordinarily disciplined.

Controversy: NetApp Outperforms EMC in SAN Database Benchmark

We just released benchmark results showing that our FAS storage systems outperform EMC’s CLARiiON on SAN database workloads. For details, see Brian Pawlowski’s blog. The quick summary is that a NetApp FAS3040 beat an EMC CX3 Model 40 on SPC-1, which is an industry standard benchmark that measures OLTP (Online Transaction Processing) performance. Our system cost less, had fewer disks, and beat the EMC by 24%. With snapshots enabled on both systems, NetApp was three times faster. Here’s the chart:

Why the controversy? EMC has never posted any SPC-1 results, so we had to run the benchmark ourselves. We did follow EMC’s best practice document for the CLARiiON, and we did have the Storage Performance Council independently audit the results. But still, the fact that we ran the tests ourselves caused concern. For instance, Chuck Hollis, the VP of Technology Alliances at EMC, raised questions about the integrity of the auditor, calling him “an infamous part-time ‘administrator’ for the Storage Performance Council” and saying that “It seems to be a part time gig for him to make a few bucks.” Chuck also raised questions about why we would do this: “The only reason you'd spend the money to buy the equipment and run the tests is to put your competitor in a bad light. I think most reasonable customers would figure that part out.”

It is fair to ask why we did this, so let me share our thinking:

First, to continue improving our SAN and database credibility. Given our NAS origins in the early 1990s, many customers and analysts have been skeptical of NetApp’s SAN capabilities. Results like this show today’s reality. Steve Duplessie at the Enterprise Strategy Group commented that “Netapp appears to have legit block performance, and shouldn't be dismissed because people (like me) presume it can't be true.” Chuck Hollis is a vocal skeptic of NetApp’s ability to play in SAN and database environments, so it shouldn’t surprise him that we want to refute his claims. (See here for a brief history of our benchmarking efforts over time.)

Second, to showcase our snapshot performance. Snapshots help customers improve backups and archive old data, and writable snapshots (FlexClones) let customers completely rethink their database test and development strategy. Unfortunately, snapshots in most storage systems are unusably slow. With NetApp, performance dropped only 3% with snapshots on. With EMC, performance dropped by a factor of three. I freely admit that our goal in focusing on snapshot performance was to “put our competitor in a bad light.” I think that’s fair because EMC’s snapshots really are painfully slow in real-world use. On the other hand, we expected our non-snapshot performance to be about the same, or maybe even a bit lower, given that our system is less expensive, has fewer disks, and uses RAID-6 instead of mirroring. Winning there was a pleasant surprise.

Third, because Chuck asked us to. In his blog entry on SPC, Chuck said: “We've never done an SPC test, and probably will never do one. Anyone is free, however, to download the SPC code, lash it up to their CLARiiON, and have at it.” I don’t promise always to follow Chuck’s advice, but I think it’s important to recognize good ideas no matter where they come from!

One key take away from this result is that turning on a simple feature like snapshots can radically change performance. Don’t let a bad experience with EMC’s snapshots scare you away from NetApp’s.

Let me close with a final word on benchmarks. Any honest vendor will agree that benchmark results are sometimes misleading, and that you should examine the details carefully. We believe that SPC-1 effectively simulates OLTP workloads, and we used real-world configurations based on each company’s own best practices documentation. But despite our best efforts, I stand by my argument in a previous blog entry that you should admire and respect great benchmark results, but also be careful.

 

 

The following SPC-1 results have been posted at www.storageperformance.org:

NetApp FAS3040 (baseline):
http://www.storageperformance.org/results/benchmark_results_spc1#a00057

NetApp FAS3040 (with Snapshots):
http://www.storageperformance.org/results/benchmark_results_spc1#a00058

EMC CLARiiON CX3 Model 40 (baseline):
http://www.storageperformance.org/results/benchmark_results_spc1#a00059

EMC CLARiiON CX3 Model 40 (with SnapView):
http://www.storageperformance.org/results/benchmark_results_spc1#a00060

All comparisons are current as of January 29, 2008.

 

Test and Development Copy of an Oracle Database: 2400% Speed Up

Niel Armstrong is the CIO of Activision, and we gave a talk together at Oracle Open World. Niel uses pretty much the full suite of Oracle products, including the database itself, much of the middleware stack, the eBusiness Suite, and Hyperion. He also uses pretty much the full suite of NetApp products.

My favorite story was about the speed up Niel saw in making copies of his production database for test and development. Before he started working with NetApp, test and dev copies were a real pain. It took a “long week” to create a full copy. (A “long week” is when you start at close of business Friday, keep working for the whole week, and don’t finish till Sunday night the weekend after. Or maybe Monday morning.) It was so painful that they did a process-re-engineering project that drove the time from roughly nine days down to four. Niel said, “We felt great about cutting the time in half. This was a big project for us, and we worked really hard at it.”

When Niel switched to NetApp storage, he started using NetApp’s copy-on-write clones (FlexClone) to make test and dev copies, and he cut the time from 4 days to 4 hours. Actually, he said it’s usually 2 hours, but he says 4 just to give himself some buffer.

This completely changed their workflow model. Niel said, “Before NetApp, we used to really fight these clones. We just didn’t want to do them. No more than one or two per large Oracle project if possible. Now we do them all the time. I’ve got over a dozen clones right now just for the UK Oracle implementation. One guy wanted a clone for a training class, and we gave it to him. We never would have done that before. It would have taken too long and used up too much storage.”

Clones are handy during normal operations, but they are especially important when you are making a major change to your environment. “Without NetApp clones, we would never have finished our upgrade in time.”

What I love most about Niel’s story is that it is almost exactly the same thing I wrote about last year after Oracle Open World (read here), except then I focused on the demo we were giving in our booth, and now I’m describing a customer who is in production and who stood on stage next to me to describe his experiences.

Let me give you a sense of Niel’s environment. Activision has game design studios all around the world. A big problem was getting reliable backups at the studios, so they implemented remote disk-to-disk copies (SnapMirror) from each studio to their disaster recovery center in Burbank, as well as replicating all of the headquarters data. A few Sundays ago, disaster recovery was “tested” by two back-to-back power failures, courtesy of SoCal Edison. They recovered in less than 4 hours, with no business interruption, which earned them attaboys from both the Chairman and the CEO.

I asked Niel whether he mostly makes clones from his DR systems, or his production systems, or what. His response: “We make clones pretty much everywhere. It’s clones all over.”

NetApp’s VTL: Best for Tape Libraries You Can Park Your Car In

We’ve been shipping our NearStore Virtual Tape Library (VTL) for about 18 months, and it’s interesting to dig into what customers actually like about it. When we first shipped, we had a good sense of the important benefits. We believed that virtual tape would help customers:

•  reduce their backup window
• reduce their restore time
• reduce the failure rate of restores

We were pretty close.

For instance, one of our most popular features is fast hardware-based compression, which lets the VTL run just as fast with compression as without. Without special compression hardware, you would still save space, but performance would be much lower. Saving money (by using less disk space) is always important, but meeting the backup window is so critical that compression without performance doesn’t cut it. (Stay tuned for VTL de-duplication, next year, which will further reduce storage costs.)

On the other hand, I’m surprised how important creating real tapes from the VTL has remained. Despite all the hoopla about disk-to-disk backups, eighty percent of VTL customers still rely on tape for some part of their process. One lesson here is how reluctant people are to eliminate processes in their “data protection path”. I expect that customers will be creating tapes and offsiting them for the next decade, so “work really well with my existing tape infrastructure” is a critical VTL requirement. What has changed is that customers go to tape much less often. With VTL, most customers go to tape weekly or less, instead of nightly, which lets them get much more mileage from their existing tape infrastructure.

The interaction between virtual tape and real tape is a fertile ground for innovation. The ability to support many virtual tapes at once, and then write real tapes later, means that you don’t have to schedule multiple jobs to the same physical tape drive. A problem with one thread doesn’t affect everything downstream, which makes a chaotic system much more deterministic. Staging backup data on disks also allows you to run tape drives at full speed -- avoiding the stop-starts that kill performance. This helps stretch existing tape infrastructure still further.

A final thing we’ve learned is that our VTL is so powerful that we tend  to compete best in large data centers. At one point we explained to our sales force, “You’re odds of winning are best if the customer has a tape library that you can park your car in.” We have introduced smaller products since then, but we need to come down even further to cover the whole market. We’re working on it.

 

Is NFS a Form of SAN? (NFS for Enterprise Apps)

I recently wrote about how both Oracle and VMware support NFS for situations that would have traditionally used block-based storage, like SAN or iSCSI.

Here is my favorite reaction to Oracle's Direct NFS Client:

This is so wrong, but I guess they're so far down the path it no longer matters.
-– Wes Felter 

I disagree that it’s “wrong”, but it’s important to ask why Oracle over NFS gives some people the heebie-jeebies. Perhaps people worry that the NFS protocol might be “chatty” – somehow less efficient at transporting data across the network. It’s true that NFS has lots of fancy requests that iSCSI and Fibre Channel SAN do not, to create directories, set permissions, move files, and so on, but none of that matters when all you do is read and write blocks of data.

At the protocol level, block-reads over NFS and block-reads over iSCSI are almost identical. The main difference is that NFS asks for a certain number of bytes, starting at a given byte offset, and iSCSI asks for a certain number of blocks, starting at a given block offset. With NFS you must divide by 512 to convert bytes to blocks. So what! The other difference is that iSCSI and FC-SAN use a LUN to identify the container holding the blocks, and NFS uses a file handle. I’m over-simplifying, but you get the point. At the protocol level, for block traffic, there is almost no difference between NFS and iSCSI or SAN.

A LUN and a file are both just containers that hold blocks of data, so get over it. It was this similarity that originally convinced me NetApp could “unify” SAN, NAS and iSCSI into a single storage appliance. (This is the key idea behind unified storage.)

There are significant differences in how the protocols connect to the host operating system. FC-SAN and iSCSI both plug in at the block layer, while NFS plugs in at the filesystem layer. Those have very different paths through the OS. Historically, bugs in the filesystem layer slowed down NFS. For instance, NFS in Solaris used to be single-threaded, so multiprocessor systems only got one CPU worth of NFS performance. Those bugs have been fixed, but in most operating systems, the block layer is still a slightly lighter-weight interface.

This is why it is so significant that both VMware and Oracle have now built NFS directly into their software as if it were a block interface. When Oracle generates NFS packets directly, it completely bypasses the OS filesystem layer, and it reduces the difference between NFS and iSCSI to the minimal protocol differences that I described above. VMware takes block level requests out of virtual machines and converts them into NFS requests, again blurring the distinction between NFS-as-NAS and NFS-as-blocks (or SAN).

Why bother converting blocks to NFS? After all, I just argued that NFS and iSCSI are almost identical for block traffic, so why not just use iSCSI if you want Ethernet, and FC-SAN if you prefer Fibre Channel?

Those extra NFS operations link to filesystem capabilities that are valuable for managing large numbers of containers. If you only had one LUN, or just a handful, it probably wouldn’t make any difference, but if you have hundreds or thousands of LUNs, it’s very convenient to give them meaningful names, group them into directories, and back them up with filesystem tools designed to handle thousands (or even millions) of separate objects.

NFS also lets you virtualize the data path. NFS provides an abstraction of the path from Application Server to Storage that customers really like and leverage. A whole industry of Fabric virtualization products is coming to market for Fibre Channel (think IBM SVC) to solve a problem already solved by NFS and TCP/IP. Path virtualization is why NFS does so well in grids. It’s the key to simple application mobility, and enables all kinds of resilience techniques.

To be clear, I’m not arguing that NFS is always the best solution for traditional block-based applications. We have many happy customers running Fibre Channel SAN and iSCSI, and we are investing aggressively in those protocols. My point is that many people simply reject NFS, mostly based on old data and misunderstandings. NFS definitely deserves consideration. Sometimes – not always – NFS is the best solution for mission critical enterprise applications.

Driving a Tesla Electric Sports Car is a Strange Form of Virtual Reality

This week, I got a chance to drive a Tesla electric roadster on the twisty mountain roads above Silicon Valley. All electric, lithium-ion, zero-to-sixty in less than four seconds – not that I stopped at 60. (Whee!)

The electric motor feels very different from an internal combustion engine. Much smoother. In my regular car, if I’m driving at 50 mph and then floor it, I get a jerk of acceleration right away, and then a second jerk an instant later, when it downshifts, and then a surge. In the Tesla, it was just continuous, smooth acceleration. This makes the car track wonderfully through curves. The jerks in a normal car sometimes threaten to shake loose the wheels if you hit the accelerator too much at the wrong time, but in the Tesla, the power is a smooth surge no matter how quickly you jam the accelerator.

In my regular car, if I suddenly let off the gas after accelerating hard, I get a reverse jerk as engine compression braking suddenly slows the vehicle. In the Telsa, no jerk at all. The acceleration stops, and then a gentle deceleration kicks in. An induction motor has no natural braking (no cylinders to compress), but it’s nice for the car to slow down when you lift off the accelerator, so they programmed in some regenerative braking. In a regular car, the amount of deceleration is a side effect of cylinder size, compression, gearing, and so on, but in the Tesla, they get to program in exactly what feels right. Turns out that people prefer just a bit of deceleration at high speeds, but at slower speeds, for street driving, people like more deceleration.

In a sense, driving a Tesla is like a strange form of virtual reality. The sensations I’m feeling are designed by a programmer who has full control of the mapping between accelerator pedal and torque. (In fact, one goal of the drive was to get feedback to the Tesla engineers on their firmware choices.)

Another example is creep, which is the way that cars with automatic transmissions creep forward slightly when you lift the brake. In a regular car, that’s a natural side effect of how the transmission works, but in a Tesla, they decided to maintain that same behavior for safety, so that you won’t get out of the car with it turned on. Lift the brake and it “reminds” you, by creeping, that it is still running.

I shouldn’t say virtual reality, because I was in a real car feeling real acceleration, but it has that same designed-by-a-programmer feeling that virtual reality does. At first I thought that designed reality might be a better description, but an internal combustion engine is certainly designed, and the acceleration and engine braking profile is something that designers take into account. I think the real difference is the level of direct control that you have with programming, as compared to the indirect influence you have when you design with physical materials.

What enables that feeling of programmed reality in the Tesla is the flat torque curve of their electric induction motor. Its torque is almost perfectly flat from 0 to 6000 rpm and then drops linearly to 50% torque at 11,000 rpm. An induction motor can generate full torque even at 0 rpm, and can go from no power to full power in milliseconds. By contrast, an internal combustion engine can’t generate any torque at 0 rpm (it stalls), and it delivers full torque only in a narrow range.

In order to generate programmed reality, you need a physical medium that is sufficiently malleable, like an induction motor or an LCD display, to give you full programmable control. Perhaps the real lesson here is that – given the flexibility of programmed systems – we should have a higher expectation of usability and design elegance for programmed reality. With great power comes great responsibility.

Why Run VMware Over NAS?

At VMWorld yesterday, I was surprised how excited customers are about using NFS to access VMDKs, even for virtual machines hosting Windows. (A VMDK is a VMware Virtual Disk, and it holds the boot image for its virtual machine.)

Since a VMDK is a virtual disk, I had assumed that block-based protocols like iSCSI and Fibre Channel would make more sense than NAS, so I asked several customers why they prefer NFS.

The answer is simple: Managing .vmdk files is much easier than managing LUNs. If you have 20 or 30 virtual machines, then VMFS is great for consolidating the VMDKs into a single LUN. But NAS is much easier and more scalable if you have hundreds or thousands of virtual machines.

The big advantage is that you can use all your file management tools. Group the VMDKs for Exchange servers in one folder, SQL servers in a second, virtual desktops in a third, and so on. Instead of backing up LUNs or virtual machines individually, simply backup a directory tree of VMDKs all at once. (This is much less expensive than buying a backup license for each virtual machine, and also easier to manage.) For disaster recovery, you can replicate the data for a whole group of virtual machines as a single unit.

Some people are surprised that you can use NFS for Windows virtual machines, since Windows can’t boot from NFS. This works because VMware has built NFS into ESX’s disk virtualization layer. ESX handles the NFS protocol so that the operating system doesn’t have to. This is very similar to Oracle’s recent project to build NFS support directly into their database.

I rarely find excuses to praise Chuck Hollis from EMC, but in a recent blog he said:

I think that, in the long term, we'll find high-end NAS much more friendly for high-end VMotion / DRS farms than today's SANs. And I think that NAS has the potential to offer a few benefits that we might not find in the SAN world.

Brilliant!

VMware’s Founder Helped To Inspire WAFL

At VMworld yesterday, I got to meet with Mendel Rosenblum, one of VMware’s founders. I want to share the story of how he helped inspire WAFL.

In the early days of NetApp, when we first started developing our WAFL file system, we drew inspiration from three main file systems: FFS, Episode and LFS:

The Berkeley Fast File System (FFS) was written by Kirk McKusick. I had worked on FFS at two prior companies (MIPS and Auspex), so I was very familiar with it.

The Episode File System was developed by Transarc, which spun out of the Andrew File System (AFS) project at Carnegie Mellon. One of the architects of Episode was Mike Kazar, who joined NetApp when we acquired Spinnaker.

The Log-structured File System (LFS) was developed as part of John Ousterhout’s Sprite operating system project at Berkeley.

The graduate student who actually designed and implemented LFS was Mendel Rosenblum. It took me quite a few years to figure out that this guy whose work I admired 15 years ago was the same guy who started VMware. Imagine my surprise!

Given that a VMware founder helped inspire WAFL, it seems there’s a sort of poetic justice that so many VMware customers use it for their data.