There is no maximum prescribed limit for your investments in securities market. However, RGESS benefits will be available only for investments in eligible securities upto Rs. 50,000.
This investment can be made in installments. However, the benefits can be availed only for the investments made in the first financial year in which the benefit is claimed under Section 80CCG.
Hence, if you are making investments in installments, make sure that you are making all your installments in a single financial year (which should be the year in which you open / designate your account for RGESS) as subsequent investments will not be counted towards RGESS.