# The Memory-Efficient Guide to Blazing Fast CSV Filtering with PHP

When it comes to processing large CSV files, memory consumption can quickly become a bottleneck. Loading a 10-million-line CSV file into memory? That's a recipe for an out-of-memory error. But what if I told you that you could **Filter** such a file using just 2MB of RAM and process it in few seconds.

Before diving into performance testing, you'll need a substantial dataset. Here's how to generate a 10-million-line CSV using the Faker library

```bash
composer require fakerphp/faker
```

[Generate dataset script whit Faker](https://gist.github.com/millancore/565bb32cb99b4e0581ac7f03dc6aaebe)

Now that we have a dataset, we must start by reading it, the most efficient way is to use a Generator which is also an Iterator, this provide elegant approach to memory-efficient CSV processing. They're perfect for processing large datasets because they maintain their state between iterations without keeping the entire dataset in memory.

```php
function readLines() : Generator
{
    $file = fopen(__DIR__.'/data.csv', 'r');

    while (($data = fgetcsv($file)) !== false) {
            yield Record::fromArray($data);
    }
    fclose($file);
}
```

Now the next key piece is Filters, PHP provides the built-in [FilterIterator](https://www.php.net/manual/en/class.filteriterator.php) class, which efficiently implements the filtering logic while maintaining memory efficiency when working with large datasets.

lets create a some filters:

```php
class AgeFilter extends FilterIterator {

    public function __construct(
        Iterator $iterator,
        private readonly int $age
    ) {
        parent::__construct($iterator);
    }

    public function accept(): bool
    {
        return $this->current()->age == $this->age;
    }
}
```

```php
class GenderFilter extends FilterIterator {

    public function __construct(
        Iterator $iterator,
        private readonly string $gender
    ) {
        parent::__construct($iterator);
    }

    public function accept(): bool
    {
        return $this->current()->gender == $this->gender;
    }
}
```

Next, we need to create a filter chain to process the previous result. To simplify this task, we can leverage the **PipelineFilterIterator** library.

```bash
composer require millancore/pipeline-iterator
```

```php
use Millancore\PipelineIterator\PipelineFilterIterator;

$iterator = PipelineFilterIterator::create(readLines())
    ->filter(GenderFilter::class, 'F')
    ->filter(AgeFilter::class, 30);

foreach($iterator as $item) {
   echo $item->name.PHP_EOL;
}

/**
* Records: 10.000.000
* File size: 227 MiB
* Time: 7.65 seconds
* Memory: 2 MB
*/
```

It's that simple! This approach ensures our filters are reusable, testable, and easily combinable to suit your needs.

### Bonus

You can use built-in Filters as [**CallbackFilterIterator**](https://www.php.net/manual/en/class.callbackfilteriterator.php) **or** [**RegexIterator**](https://www.php.net/manual/en/class.regexiterator.php)

```php
 # Regex filter names containing 'ana'
 ->filter(RegexIterator::class, '/ana/')
```

> To use RegexIterator the class must implement \_\_toString to apply the filter.

### Thanks