This page describes what x264 settings I think look best with anime (or any cartoon). It’s written with the assumption that the reader already knows how to encode and what programs to use. This isn't a step-by-step tutorial so if you're new to encoding then you'll probably want to start elsewhere.
Which settings are important
If you’re unfamiliar with or don’t fully understand the different x264 options and what they do just remember that almost always you’ll get better results with the more options you enable and the higher the numbers are. This isn’t always the case though, and the two important things that need to be custom for your video source are the deblocking and custom quantization matrices (CQM). The default deblocking settings are 0,0 which should make most video sources look decent. Without going into what exactly deblocking does, I’ll just say that anime will almost always look much better with higher, but not too high, settings. It’s generally recommended to use 1,1 although I personally think 1,2 looks better. DO NOT use negative values. They bring out more details but they also make solid colors look awful. CQMs from what I’ve found are mostly useless with anime so I’d just leave that option turned off. If you really want to use one though I’d recommended one of *.mp4 guy’s.
Another important, although much simpler, setting is the bitrate. Bitrate isn't very complicated but I'll go into it just because its worth mentioning. A popular bitrate is 1000KBps which most of the time will achieve the middle ground between file size and quality. If you’re using an action packed video then you’ll probably want to use a higher bitrate since action and colors aren’t as compressible as a quiet slice of life video. Also you should always use 2 pass encoding, if you don’t like how much longer that takes to encode then reduce the settings on the first pass. Lots of GUIs can automate this for you. Note that x264 also supports things like constant quality rather than constant bitrate but I haven’t found that as satisfying. Remember though that the filesize of the output file is directly related to the bitrate you use, but so is the quality!
Remember that a lot of settings will technically make your video look better, but can add incredible lengths of time to the encoding process. Often it's not worth using them. An example would be enabling "exhaustive search" for Motion Estimation (--me esa). You will very rarely notice any difference it makes in the output but takes incredibly longer to encode. Using "uneven multi-hexagon" (--me umh) is a much wiser choice. I've included the settings I use below which I think do the best at maximizing the quality while minimizing encoding time.
My settings
Here are the command line settings I use (last used on the 680 build of x264):
First pass:
--pass 1 --bitrate 1000 --stats ".stats" --bframes 3 --b-pyramid --filter 1,2 --subme 1 --analyse none --vbv-maxrate 2500 --me dia --threads auto --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --output NUL "path/to/avs/script.avs"
second pass:
--pass 2 --bitrate 1000 --stats ".stats" --ref 5 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --bime --weightb --filter 1,2 --trellis 1 --analyse all --8x8dct --vbv-maxrate 2500 --me umh --threads auto --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --output "path/to/output/file.264" "path/to/avs/script.avs"
And here are my settings in a meGUI profile file.
How did I chose these settings? Basically I just used trial and error to find out what looked best and was most efficient on my machine. I found that some options would make the encoding process take hours longer to complete with no noticeable increase in quality at all (even comparing two still frames showed hardly any difference) so I chose not to use those options. One thing that wasn’t a concern to me was compatibility. These settings won’t play on QuickTime, iPods, and probably on plenty of other software and hardware that claim to play h.264. I just wanted to make the video look as good as I could, and most people use programs like ffdshow which will support these settings anyway.
My settings in action
If you want to see some video examples here are some torrents for videos I encoded using these settings, give or take 300-1000KBps.
And here are some still screenshots to show how compression effects different visual elements. They were taken from a video encoded using the settings shown above with the 680 build of x264. You can move your mouse over them to view the original DVD source for comparision. These images are losslessly compressed PNGs and the only tampering done to them was a bilinear resize to preserve the original aspect ratio. (The DVD had a resolution of 720x480 with a PAR of 4:3, these images are now 640x480 to preserve that ratio).
Conclusion
When you do your own encoding you should do plenty of test runs with different settings to see what looks the best for you and your video source. If you have any questions I’ll be happy to answer them but you’ll probably find a lot more help if you look around the doom9 forums.