merge in LiipFunctionalTestBundle

[lsmith77]

i wasnt quite happy with the way this Bundle was born, the fork was unnecessary if there would have been some patience to collaborate. anyway, today LiipFunctionalTestBundle has quite a user base but development is quite slow. ICBaseTestBundle provides several very useful additional features. as i dont really see much use in 2 bundles in this area, i am willing to deprecate LiipFunctionalTestBundle in favor of this Bundle.

that being said I suspect especially in regards to fixture loading there might be some useful things that are only available in LiipFunctionalTestBundle. maybe a few other things here and there. not sure if anyone is using the html5 validation service integration, so i am willing to loose that along the way.

what do you think?

[guilhermeblanco]

@lsmith77 By no means this bundle was created with the purpose of becoming a competitor with LiipFunctionalTestBundle. Quite the opposite. We started using he bundle and some portions of our codebase are still using,
What happened was a couple of subsequent issues and small breaks that lead us to have more control over what we were trying to do, allowing extensibility and also provide a better way to manage fixtures. That was the origin of our bundle. We haven't even considered to open source it initially.

But here we are today and right now we're spending more and more efforts on this bundle. We spent a couple of days detailing how to test individual pieces from Unit and Functional perspective. We're expanding what we originally have defined to accommodate not only what we wrote an year ago, but also more and more granular helpers to keep testing simpler.

Based on that, we'd be more than happy to include whatever feature you consider missing. We're still playing with a "new" version to deal with Unit testing, and as soon as it gets finished, we'll enter on Functional testing perspective.
I'd be interested to understand what are the features we could implement, always keeping in mind we're not concentrating our efforts right now on Functional perspective.
My main goal is to have an ability to use sqlite:memory as DB at the end, but I have no idea how I'd do this for now, which would require further evaluation. Also, you should be aware that Doctrine team is investing some time to release a new major version of DoctrineFixturesBundle, which should be available as part of Symfony 2.3.

Let's keep this conversation going, I'm interested on feature we could bring to simplify everyone's life.

[lsmith77]

I am not sure if sqlite:memory is so important. There is an open PR for LiipFunctionalTestBundle liip/LiipFunctionalTestBundle#85 which will fully automate the resetting of the cached sqlite DBs. Obviously in memory could be faster than on disc, but at least this would ensure that caching can be used without any annoyances.

I do not have a perfect overview of the features of this Bundle, nor do I really know all the features of LiipFunctionalTestBundle in detail, but a rough overview of LiipFunctionalTestBundle would be:

• running commands
• getting the container with support for multiple kernels in the app
• loading fixtures for ORM/ODM
• for ORM additionally support loading a schema and caching the sqlite DB for ORM
• creating a pre-authenticated client instance
• helper methods to validate and fetch response content
• integration with an html5 validation service (this feature can be dropped)
• ability to automatically fail functional tests if too many queries are executed

[guilhermeblanco]

Hi,

I'll add feedback to each one point...

I am not sure if sqlite:memory is so important. There is an open PR for LiipFunctionalTestBundle liip/LiipFunctionalTestBundle#85 which will fully automate the resetting of the cached sqlite DBs.

We partially fixed this issue by holding an MD5 of your sorted Fixture list. We create a cache for each combination, which prevents some of this from happening in some cases. We're still unable to fix the issue where your Fixture list remains the same and you only include new entries. That still requires a cache clear, or a sqlite:memory to be configured.

The problem that we face affects our entire infrastructure. We rely on ESIs a lot for individual components. A page is built of multiple sub-requests. Symfony sends a new request for each one of them and since the DB is configured to use sqlite:memory, it would create a new instance of DB for the sub-request which would be always blank, then failing the tests.
I don't know how we're gonna solve this issue, but it's the blocker that InstaClick is facing to use memory as DB.

running commands

We already support the execution of commands through a Helper.
https://github.com/instaclick/ICBaseTestBundle#command-helper

getting the container with support for multiple kernels in the app

We didn't see a need to have multiple kernels being loaded on a single test, but we identified the need to switch the Kernel for a given test. We rely on Symfony's default support for that:
https://github.com/symfony/symfony/blob/master/src/Symfony/Bundle/FrameworkBundle/Test/WebTestCase.php#L112

loading fixtures for ORM/ODM

We haven't included any support for ODM so far. After Doctrine's refactoring on Fixtures, it would be a simple execution for this. Please refer to new branches of Fixtures and its own Bundle:

• Library: https://github.com/doctrine/data-fixtures/tree/2.0
• Bundle: https://github.com/doctrine/DoctrineFixturesBundle/tree/3.0
  Code is still not complete on Bundle, but the library is already stable. We'll modify support to be compliant with this new version as soon as it gets released. Loading fixtures in this new version for ORM, ODM and/or PHPCR consumes same API.

for ORM additionally support loading a schema and caching the sqlite DB for ORM

We support 2 levels of caching for SQLite DB.

• The first one caches the schema creation. It only gets executed once and after that it gets copied over and over until your schema changes or you clear your cache.
• It caches content. We replicated the same concept of LiipFunctionalTestBundle here. We get a created schema and we load with data from fixtures. The difference is that we also cache the ReferenceRepository, allowing consumers to refer to specific instance values (instances are detached Entities, not being managed by Doctrine) without requiring to search for them, slowing down the entire testing infrastructure.

creating a pre-authenticated client instance

This is also supported through a Helper.
https://github.com/instaclick/ICBaseTestBundle#session-helper

integration with an html5 validation service (this feature can be dropped)

We haven't identified any need for this so far. Easy to be included.

ability to automatically fail functional tests if too many queries are executed

This is a very interesting feature that I'm really interested to bring in. We have our SLAs that are not being monitored in an automated fashion and that would be a great solution for this problem.
I'll investigate how we could achieve it in a clean way and merge into our codebase.

[lsmith77]

Thank you for the detailed response.

A few short comments:

1. the idea with Remove Cached Databases on Test Finish liip/LiipFunctionalTestBundle#85 is to not only check the md5 from the fixture list but also check the change time of the files and compare it to the creation date of the cached db
2. ODM support was actually quite easy to add

At any rate, since I am short on time I will likely not have time to do an in depth analysis here but it sounds to me like what is missing is pretty small and can likely be added if needed. So I might just announced that LiipFunctionalTestBundle is deprecated. However ODM support is quite critical for us, so this is something that we would urgently need to see added.

[lsmith77]

here is a PR to add ODM support #28

[havvg]

What's the current status on this discussion, any movement happening or will there be both bundles?