Ytukyg

You've several options:

1. 
   

   Remove some of the bias.

   
   
   

   (a) By penalizing the likelihood as per @Nick's suggestion. Package logistf in R or the FIRTH option in SAS's PROC LOGISTIC implement the method proposed in Firth (1993), "Bias reduction of maximum likelihood estimates", Biometrika, 80,1.; which removes the first-order bias from maximum likelihood estimates. (Here @Gavin recommends the brglm package, which I'm not familiar with, but I gather it implements a similar approach for non-canonical link functions e.g. probit.)

   
   
   

   (b) By using median-unbiased estimates in exact conditional logistic regression. Package elrm or logistiX in R, or the EXACT statement in SAS's PROC LOGISTIC.

   
   



2. Exclude cases where the predictor category or value causing separation occurs. These may well be outside your scope; or worthy of further, focused investigation. (The R package safeBinaryRegression is handy for finding them.)



3. 
   

   Re-cast the model. Typically this is something you'd have done beforehand if you'd thought about it, because it's too complex for your sample size.

   
   
   

   (a) Remove the predictor from the model. Dicey, for the reasons given by @Simon: "You're removing the predictor that best explains the response".

   
   
   

   (b) By collapsing predictor categories / binning the predictor values. Only if this makes sense.

   
   
   

   (c) Re-expressing the predictor as two (or more) crossed factors without interaction. Only if this makes sense.

   
   



4. Use a Bayesian analysis as per @Manoel's suggestion. Though it seems unlikely you'd want to just because of separation, worth considering on its other merits.The paper he recommends is Gelman et al (2008), "A weakly informative default prior distribution for logistic & other regression models", Ann. Appl. Stat., 2, 4: the default in question is an independent Cauchy prior for each coefficient, with a mean of zero & a scale of $frac{5}{2}$; to be used after standardizing all continuous predictors to have a mean of zero & a standard deviation of $frac{1}{2}$. If you can elucidate strongly informative priors, so much the better.




5. Do nothing. (But calculate confidence intervals based on profile likelihoods, as the Wald estimates of standard error will be badly wrong.) An often over-looked option. If the purpose of the model is just to describe what you've learnt about the relationships between predictors & response, there's no shame in quoting a confidence interval for an odds ratio of, say, 2.3 upwards. (Indeed it could seem fishy to quote confidence intervals based on unbiased estimates that exclude the odds ratios best supported by the data.) Problems come when you're trying to predict using point estimates, & the predictor on which separation occurs swamps the others.




6. Use a hidden logistic regression model, as described in Rousseeuw & Christmann (2003),"Robustness against separation and outliers in logistic regression", Computational Statistics & Data Analysis, 43, 3, and implemented in the R package hlr. (@user603 suggests this.) I haven't read the paper, but they say in the abstract "a slightly more general model is proposed under which the observed response is strongly related but not equal to the unobservable true response", which suggests to me it mightn't be a good idea to use the method unless that sounds plausible.




7. "Change a few randomly selected observations from 1 to 0 or 0 to 1 among variables exhibiting complete separation": @RobertF's comment. This suggestion seems to arise from regarding separation as a problem per se rather than as a symptom of a paucity of information in the data which might lead you to prefer other methods to maximum-likelihood estimation, or to limit inferences to those you can make with reasonable precision—approaches which have their own merits & are not just "fixes" for separation. (Aside from its being unabashedly ad hoc, it's unpalatable to most that analysts asking the same question of the same data, making the same assumptions, should give different answers owing to the result of a coin toss or whatever.)

This is an expansion of Scortchi and Manoel's answers, but since you seem to use R I thought I'd supply some code. :)

I believe the easiest and most straightforward solution to your problem is to use a Bayesian analysis with non-informative prior assumptions as proposed by Gelman et al (2008). As Scortchi mentions, Gelman recommends to put a Cauchy prior with median 0.0 and scale 2.5 on each coefficient (normalized to have mean 0.0 and a SD of 0.5). This will regularize the coefficients and pull them just slightly towards zero. In this case it is exactly what you want. Due to having very wide tails the Cauchy still allows for large coefficients (as opposed to the short tailed Normal), from Gelman:

How to run this analysis? Use the bayesglm function in arm package that implements this analysis!

library(arm)



set.seed(123456)

# Faking some data where x1 is unrelated to y

# while x2 perfectly separates y.

d <- data.frame(y  =  c(0,0,0,0, 0, 1,1,1,1,1),

                x1 = rnorm(10),

                x2 = sort(rnorm(10)))



fit <- glm(y ~ x1 + x2, data=d, family="binomial")



## Warning message:

## glm.fit: fitted probabilities numerically 0 or 1 occurred 



summary(fit)

## Call:

## glm(formula = y ~ x1 + x2, family = "binomial", data = d)

##

## Deviance Residuals: 

##       Min          1Q      Median          3Q         Max  

## -1.114e-05  -2.110e-08   0.000e+00   2.110e-08   1.325e-05  

## 

## Coefficients:

##               Estimate Std. Error z value Pr(>|z|)

## (Intercept)    -18.528  75938.934       0        1

## x1              -4.837  76469.100       0        1

## x2              81.689 165617.221       0        1

## 

## (Dispersion parameter for binomial family taken to be 1)

## 

##     Null deviance: 1.3863e+01  on 9  degrees of freedom

## Residual deviance: 3.3646e-10  on 7  degrees of freedom

## AIC: 6

## 

## Number of Fisher Scoring iterations: 25

Does not work that well... Now the Bayesian version:

fit <- bayesglm(y ~ x1 + x2, data=d, family="binomial")

display(fit)

## bayesglm(formula = y ~ x1 + x2, family = "binomial", data = d)

##             coef.est coef.se

## (Intercept) -1.10     1.37  

## x1          -0.05     0.79  

## x2           3.75     1.85  

## ---

## n = 10, k = 3

## residual deviance = 2.2, null deviance = 3.3 (difference = 1.1)

Super-simple, no?

References

Gelman et al (2008), "A weakly informative default prior distribution for logistic & other regression models", Ann. Appl. Stat., 2, 4
http://projecteuclid.org/euclid.aoas/1231424214

One of the most thorough explanations of "quasi-complete separation" issues in maximum likelihood is Paul Allison's paper. He's writing about SAS software, but the issues he addresses are generalizable to any software:

• Complete separation occurs whenever a linear function of x can generate perfect predictions of y




• Quasi-complete separation occurs when (a) there exists some coefficient vector b such that bxi ≥ 0 whenever yi = 1,
  and bxi ≤ 0* whenever **yi = 0 and this equality holds for at
  least one case in each category of the dependent variable. In other
  words in the simplest case, for any dichotomous independent variable
  in a logistic regression, if there is a zero in the 2 × 2 table formed
  by that variable and the dependent variable, the ML estimate for the
  regression coefficient does not exist.

Allison discusses many of the solutions already mentioned including deletion of problem variables, collapsing categories, doing nothing, leveraging exact logistic regression, Bayesian estimation and penalized maximum likelihood estimation.

http://www2.sas.com/proceedings/forum2008/360-2008.pdf

For logistic models for inference, it's important to first underscore that there is no error here. The warning in R is correctly informing you that the maximum likelihood estimator lies on the boundary of the parameter space. The odds ratio of $infty$ is strongly suggestive of an association. The only issue is that two common methods of producing tests: the Wald test and the Likelihood ratio test require an evaluation of the information under the alternative hypothesis.

With data generated along the lines of

x <- seq(-3, 3, by=0.1)

y <- x > 0

summary(glm(y ~ x, family=binomial))

The warning is made:

Warning messages:

1: glm.fit: algorithm did not converge 

2: glm.fit: fitted probabilities numerically 0 or 1 occurred 

which very obviously reflects the dependence that is built into these data.

In R the Wald test is found with summary.glm or with waldtest in the lmtest package. The likelihood ratio test is performed with anova or with lrtest in the lmtest package. In both cases, the information matrix is infinitely valued, and no inference is available. Rather, R does produce output, but you cannot trust it. The inference that R typically produces in these cases has p-values very close to one. This is because the loss of precision in the OR is orders of magnitude smaller that the loss of precision in the variance-covariance matrix.

Some solutions outlined here:

Use a one-step estimator,

There is a lot of theory supporting the low bias, efficiency, and generalizability of one step estimators. It is easy to specify a one-step estimator in R and the results are typically very favorable for prediction and inference. And this model will never diverge, because the iterator (Newton-Raphson) simply does not have the chance to do so!

fit.1s <- glm(y ~ x, family=binomial, control=glm.control(maxit=1))

summary(fit.1s)

Gives:

Coefficients:

            Estimate Std. Error z value Pr(>|z|)    

(Intercept) -0.03987    0.29569  -0.135    0.893    

x            1.19604    0.16794   7.122 1.07e-12 ***

---

Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

So you can see the predictions reflect the direction of trend. And the inference is highly suggestive of the trends which we believe to be true.

perform a score test,

The Score (or Rao) statistic differs from the the likelihood ratio and wald statistics. It does not require an evaluation of the variance under the alternative hypothesis. We fit the model under the null:

mm <- model.matrix( ~ x)

fit0 <- glm(y ~ 1, family=binomial)

pred0 <- predict(fit0, type='response')

inf.null <- t(mm) %*% diag(binomial()$variance(mu=pred0)) %*% mm

sc.null <- t(mm) %*% c(y - pred0)

score.stat <- t(sc.null) %*% solve(inf.null) %*% sc.null ## compare to chisq

pchisq(score.stat, 1, lower.tail=F)

Gives as a measure of association very strong statistical significance. Note by the way that the one step estimator produces a $chi^2$ test statistic of 50.7 and the score test here produces a test statistic pf 45.75

> pchisq(scstat, df=1, lower.tail=F)

             [,1]

[1,] 1.343494e-11

In both cases you have inference for an OR of infinity.

, and use median unbiased estimates for a confidence interval.

You can produce a median unbiased, non-singular 95% CI for the infinite odds ratio by using median unbiased estimation. The package epitools in R can do this. And I give an example of implementing this estimator here: Confidence interval for Bernoulli sampling

Be careful with this warning message from R. Take a look at this blog post by Andrew Gelman, and you will see that it is not always a problem of perfect separation, but sometimes a bug with glm. It seems that if the starting values are too far from the maximum-likelihood estimate, it blows up. So, check first with other software, like Stata.

If you really have this problem, you may try to use Bayesian modeling, with informative priors.

But in practice I just get rid of the predictors causing the trouble, because I don't know how to pick an informative prior. But I guess there is a paper by Gelman about using informative prior when you have this problem of perfect separation problem. Just google it. Maybe you should give it a try.

I am not sure that I agree with the statements in your question.

I think that warning message means, for some of the observed X level in your data, the fitted probability is numerically 0 or 1. In other words, at the resolution, it shows as 0 or 1.

You can run predict(yourmodel,yourdata,type='response') and you will find 0's or/and 1's there as predicted probabilities.

As a result, I think it is ok to just use the results.

I understand this is an old post, however I will still proceed with answering this as I have struggled days with it and it can help others.

Complete separation happens when your selected variables to fit the model can very accurately differentiate between 0’s and 1’s or yes and no. Our whole approach of data science is based on probability estimation but it fails in this case.

Rectification steps:-

1. Use bayesglm() instead of glm(), when in case the variance between the variables is low




2. At times using (maxit=”some numerical value”) along with the bayesglm() can help

3.Third and most important check for your selected variables for the model fitting, there must be a variable for which multi collinearity with the Y (outout) variable is very high, discard that variable from your model.

As in my case I had a telecom churn data to predict the churn for the validation data. I had a variable in my training data which could very differentiate between the yes and no. After dropping it I could get the correct model. Further more you can use stepwise(fit) to make your model more accurate.